DOCUMENT RESUME 



ED 427 240 



CE 078 162 



AUTHOR 

TITLE 

INSTITUTION 

SPONS AGENCY 

PUB DATE 
NOTE 
CONTRACT 
PUB TYPE 
EDRS PRICE 
DESCRIPTORS 



Merrifield, Juliet 

Contested Ground: Performance Accountability in Adult Basic 
Education. NCSALL Reports #1. Research Briefs. 

National Center for the Study of Adult Learning and 
Literacy, Boston, MA. 

Office of Educational Research and Improvement (ED) , 
Washington, DC. 

1998-08-00 

5p.; For the full report, see ED 423 450. 

R309B60002 

Reports - Descriptive (141) 

MFOl/PCOl Plus Postage. 

♦Accountability; *Adult Basic Education; Evaluation 
Criteria; Evaluation Methods; *Outcomes of Education; 
♦Performance Based Assessment; Program Evaluation; Test 
Construction 



ABSTRACT 



This brief suggests that performance accountability offers 
the chance to refocus on what learners accomplish by participating in adult 
basic education (ABE) and to reorient every aspect of the system to achieve 
the best results. The 1966 Adult Education Act provides little guidance in 
developing performance accountability. Recent research describes literacy as 
multiple "literacies" rooted in particular social contexts. This change in 
definition shifts the focus even further from abstract skills to real-life 
practices. This conceptual change requires a revision of what performance 
means. Other issues include the following: definition of the purpose of 
literacy, accountability, fragmentary data on performance, and measurement 
tools that can collect more reliable accountability data. New initiatives are 
addressing performance accountability: the National Institute for Literacy's 
Equipped for the Future project. National Outcomes Reporting System, and 
other state and national level initiatives. Over the last few years, many 
states have been focusing on building capacity for program delivery. High 
performance or "learning organizations" from the business world may prove 
useful models for ABE. Four principles would enable the field to perform 
effectively and be held accountable for performance: agree on performances, 
develop mutual accountability relationships, build capacity to perform and be 
accountable, and create new tools to measure performance. This framework for 
future policy provides guidelines for future action. (YLB) 
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Contested Ground: Performance Accountability in 
Adult Basic Education 

Juliet Merrilield 

Center for Literacy Studies, University of Tennessee 



dult basic education (ABE) has long been viewed by many educators and policymakers as a tool for addressing social and 
economic problems. Now, in a context of global economic restructuring, changes in work and employment, and the largest 
immigration to the USA since the early 1900s, ABE must demonstrate its success in terms of student and societal outcomes. In 
short, ABE is facing demands to be accountable for its performance. 



A focus on results is new for adult education, and 
potentially transforming. Performance accountability offers 
the chance to re-focus on what learners accomplish by 
participating in ABE and to re-orient every aspect of the 
system to achieve the best results. 

It is tempting to rush into developing measurement and 
reporting systems. Experiences with performance 
accountability show that this would be a mistake. Agreement 
on what to measure must be established first. 

This policy paper lays out key issues in performance 
accountability and presents recommendations for policy and 
action. It is based on the literature from education, 
government, management, and other fields, and draws on 
interviews with researchers and adult education leaders at state 
and national levels. Its recommendations were informed by a 
discussion with a group of experienced adult education 
practitioners and researchers. 



why change? What is the problem? 

Past policy is no longer a guide for action. The 1966 
Adult Education Act, which has governed ABE for more than 
30 years, provides little guidance in developing performance 
accountability. The act charges recipients of funding with 
enabling adults to ‘'acquire the basic educational skills 
necessary for literate functioning.*’ Definitions of literate 
functioning vary and are hotly debated. Over time, views of 
what literacy means have shifted from academic skills such as 







decoding text, to functional skills, such as being able to 
perform certain tasks using literacy skills. Recent research 
has changed radically our understanding of literacy. Literacy 
is now described as multiple “literacies” rooted in particular 
social contexts. This change in definition shifts the focus 
even further from abstract skills to real-life practices. 

This conceptual change requires a re-vision of what 
“performance” means. When literacy meant mastery of what 
was taught in schools, performance was testable. When 
literacy shifted to the notion of functional competency - 
being able to perform certain tasks using literacy skills - the 
issue of performance became more complex. Tests had to 
identify which real-life literacy tasks should be included and 
which not, with no theory to guide which to choose or how to 
create scales of difficulty. Test developers had to assume that 
performance on the test equated with how well the student 
performed the real-life equivalent task. The recent research 
on literacy in its social context has been carried out through 
careful observations of literacy events and activities which 
shed light on prevailing literac^practices. While it shifts the 
focus to performance in life, not in test situations, this new 
research has not yet been incorporated into practice, 
assessment, or policy. 

Agreement on what “literate functioning” means is . 
crucial to accountability. Is literacy a right or a necessity for 
the good life? Should performance be demonstrated in terms 
of literacy skills - the earlier view - or literacy practices, the 
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current thinking? 

The purpose of literacy is not defined. The Adult 
Education Act of 1966 had broad social intent. In addition to 
not strictly defining what literacy was, it did not define the 
purpose of literacy. If a performance accountability system is 
to measure only literacy gain, the task of developing such a 
system would be challenging enough. If a system is to 
measure the achievement of literacy for a predefined purpose, 
a lack of clear objectives makes accountability systems even 
more difficult to develop. The debate in this area centers 
around whether the purpose of literacy education is individual 
advancement or community development. Are the desired 
outcomes productive workers, good citizens, or merely more 
literate people? Once these questions are answered, the next 
set of questions arise: who defines whether these outcomes 
are achieved, and how much responsibility should the literacy 
program be asked to take for these outcomes? 

Stakeholders are not mutually accountable. Another area 
of concern lies in the mutual responsibility for adult basic 
education. Many possible stakeholders - learners and 
teachers, administrators and policymakers, funders, 
employers, public school personnel, and taxpayers -* may be 
said to have a legitimate concern with the outcomes of adult 
literacy education. All stakeholders are not, however, equal in 
terms of access to information or ability and power to hold the 
adult education system accountable. Learners, for example, 
often have limited information and little power to change the 
system. Congressional representatives stand for taxpayers in 
exercising accountability over the adult education system 
which is supported by public money. Legislators are often not 
held accountable by learners or educators for providing 
adequate resources and policy guidance to the system. 

Capacity is weak. Adult basic education is struggling to 
create a national accountability system without a national 
service delivery system. It is difficult to have a management 
information system when there is no management system. 

Research reveals a fragmented and incomplete system 
with multiple funding sources and reporting formats, diverse 
institutions, competing objectives, and missing or unreliable 
performance data. In most states, staff are part-time, and 
volunteers continue to have an essential role in student 
services. Per-student funding is low, and most programs are 
not able to meet client needs for childcare or transportation. 
While data on performance are fragmentary, what there are 
suggest that most learners do not stay long, make some initial 
learning gains, but may not make long-term skill gains. 

The capacity to perform - to achieve desired goals - is 
linked with the capacity to be accountable ~ to document 
achievements and measure results. Some states are beginning 
to use program performance data successfully to improve 
program services. More reliable accountability data are 



collected when they are used at the program level to meet 
program needs. 

Measurement tools are not up to the task. Learning is at 
the heart of ABE, and its measurement is of particular concern 
to performance accountability. Adult education cannot be 
accountable to learners or to policymakers without the ability 
to track learning of individuals, to demonstrate what has been 
learned, to compare learning across programs, and to judge 
learning against external standards. 

Yet standardized tests, the most widely used tools for 
measuring learning, have been criticized both by researchers 
and practitioners because they do not demonstrate what has 
been learned. They are also incompatible with new research- 
based conceptions of literacy as social practices rather than 
isolated skills. Some programs are using various “authentic 
assessment” tools, such as portfolios, but so far these cannot 
compare learning between learners and across programs. 
Without external standards or criteria, authentic assessment 
will not meet policy needs. 

What change is happening? 

New initiatives are addressing performance 
accountability. The problems of accountability are well 
recognized and are beginning to be addressed at national and 
state levels. 

Equipped for the Future: The National Institute for 
Literacy’s Equipped for the Future (EFF) project is a broad- 
based system reform effort that has actively sought input from 
a wide range of stakeholders. These include stakeholders 
outside the adult education system (policymakers, employers, 
and civic leaders) as well as within it (adult learners, teachers, 
program administrators, and researchers). EFF has developed 
and validated a set of four purposes for adult education and 
lifelong learning: to have access to information, to give voice 
to ideas and opinions, to solve problems, and to be able to 
continue learning. EFF has related these purposes to the three 
adult roles of worker, citizen, and family member. With 25 
development partners across the country, EFF is now testing a 
set of common activities and defined skills which will form 
the basis for future development of standards and 
performance measures. When fully articulated and validated, 
these could provide a framework for performance as the basis 
for an accountability system. 

National Outcomes Reporting System: This project is 
funded by the U.S, Department of Education and has involved 
many state ABE directors. It is developing a common set of 
outcomes for adult basic education as a basis for data 
collection and reporting. However, the process has so far had 
limited input from the full range of stakeholders. 

Other state and national level initiatives: A number of 
initiatives at state and national levels are bringing stakeholder 
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groups into closer relationships. A National Summit, proposed 
by the National Center for the Study of Adult Learning and 
Literacy (NCS ALL), in partnership with NIFL and the 
Department of Education, is an exampje of one way to ensure 
stakeholder involvement in deliberations about the future of 
ABE at the national level. Local and state adult learner 
organizations are forming, increasing learner engagement 
around issues like state and federal funding. A national adult 
learner organization, VALUE (Voices for Adult Literacy 
United for Education), has recently been formed and shows 
promise for building a strong and effective student voice. 

State capacity-building: Over the last few years, many 
states have been focusing on building capacity for program 
delivery. Some have worked specifically on building capacity: 
for accountability as well. The cases of Arkansas, 

Connecticut, and Pennsylvania, reviewed in this policy paper, 
as well as other states like Massachusetts, Iowa, and 
California, all demonstrate the importance of long-term and 
systematic investment in training, information, and technical 
assistance. 

Lessons from the business world: High performance or 
“learning organizations” from the business world may prove 
useful models for adult education. These emphasize 
continuous improvement and learning in all parts of the 
organization, responsiveness to internal and external 
customers, participation in decisions, and shared 
responsibility at all levels. Traditional models of 
accountability are linear - quality control inspectors check 
widgets at the end of the production line to see if they meet 
specifications. Mutual accountability engages members of the 
organization in developing common vision, determining goals 
and customer expectations, and designing effective means of 
monitoring, producing, and improving. 

what next? A Framework for Action 

The policy paper recommends four principles which 
would enable the field to perform effectively and to be held 
accountable for performance. This framework for future 
policy builds on what has been learned about performance 
accountability in past experiences, and provides guidelines for 
future action. The principles are: 

1. Agree on performance. 

2. Build mutual accountability relationships. 

3. Develop capacity both to perform and be 
accountable. 

4. Create new tools to measure performance. 

1 . Agree on performances 

Good performance - what needs to be measured - is not 
a technical question, but inevitably in the realm of values. 



The challenge is to come to agreement on performance as a 
“big tent” which can include the full diversity of purposes. 

, Experience from business and industry suggests that it is- 
crucial that performance be defined neither too tightly nor too 
loosely. If performance is defined too tightly, a mis-match 
between system goals and individual goals may occur. For 
example, learners’ purposes for entering literacy education 
may be to read to their children, but the system may only 
measure whether they get a GED. If performance is framed 
too loosely, no shared mission or common accountability 
measures can be developed. Getting it right requires a broad- 
based and inclusive process involving multiple stakeholders. 

Lessons from the literature and experience in education 
and other fields suggest: ■ 

□ Don’t assume the question of what performance means can 
be skipped over or rushed. Without knowing what is 
important, measurement becomes an exercise in “gaming the 
numbers” to satisfy external demands, often with perverse 
results. 

□ Involve stakeholders and seek consensus. Without broad 
public debate it is difficult to frame performance goals which 
reflect the “big tent.” 

□ Reflect newer understandings of literacy, and connect 
performance with real life. This is an opportunity for literacy, 
research to connect with and support practice. 

□ Acknowledge multiple performances. Too narrow or tight a 
definition of goals will exclude learners and programs or force 
them to falsify their data. 

2. Develop mutual accountability 
relationships 

Reforming accountability in high performance terms 
requires a switch from one-way, top-down lines of 
accountability to a mutual web of accountability relationships. 
To participate, st^eholders need information and the ability 
or power to hold others accountable. This entails greater 
transparency in information, increased flows of information, 
and room at the table for groups who have not been there in 
the past. 

Bring the full range of stakeholder groups into the 
process - including teachers and learners who often have not 
been at the table. 

Provide support for stakeholders who have least access to 
information and power, including adult learner organizations 
at national and state levels. 

Increase information flows among and between all 
stakeholders. 

Develop learning organizations at the program and state 
levels which would emphasize learning and continuous 
improvement, shared responsibility, and engagement in 
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monitoring results, 

3« Build capacity to perform and be 
accountable 

State experiences with capacity-building reviewed for 
this policy paper indicate that the two kinds of capacity - to 
perform and to be accountable - are linked. By developing a 
learning organization approach in which there are continuous 
feedback loops, performance data can help programs improve 
performance. 

Build the capacity to perform. Key elements include 
increased resources, focusing resources on quality rather than 
quantity, staff development and training, technical support, 
use of performance data for continuous improvement. 

Build the capacity to be accountable. Key elements 
include accountability demands which are commensurate with 
resources and capacity, engagement of users in developing 
better measurement tools, staff training and support, timely 
information loops, rewards for improved performance, 

4* Create new tools to measure performance 

Accountability systems must meet the different 
information needs of different stakeholders. To do so, data 
users and data providers must be in communication so that the 
most appropriate measurement tools can be applied. 

Accountability systems commonly use several types of 
indicators to track performance over time. Input indicators 
provide information about the capacity of the system and its 
programs. Process indicators track participation in programs 
to see whether different educational approaches produce 
different results. Output indicators are short-term measures of 
results, and outcome indicators are long-term measures of 
outcomes and impacts. No single indicator can suffice to 
measure performance, especially of an enterprise as complex 
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as adult basic education. 

ABE invested a great deal of work into developing 
indicators of program quality, but much less on performance 
measures. New approaches and tools for measurement are 
needed which are linked to performance. 

Develop external standards or criteria against which 
individual student learning can be measured and through 
which program performance can be assessed. 

Develop performance assessment tools for measuring 
learning. These directly assess learners’ performance in terms 
of literacy practices rather than the indirect approaches of 
standardized tests which “stand for” real-life practices, usually 
inadequately. Initiatives in performance assessment in 
countries such as Britain and Australia may provide useful 
models for measuring and assessing learning. 

Use the full range of potential of research, evaluation, and 
monitoring technologies to meet the needs of different 
stakeholders. These approaches to gathering, analyzing, and 
using information are based on different kinds of data and 
meet different purposes. Using them in appropriate ways, 
adult education can develop a dynamic system of information, 
analysis, and reporting. 

Next steps 

This framework of principles for action acknowledges 
that there are no quick answers. To put the principles into 
action requires consultation with the field and with 
stakeholders. It will need meetings and taskforces, and it will 
take time. It requires learning lessons from elsewhere when 
appropriate, building on current initiatives when they are 
under way, and creating new tools when none exist. 
Policymakers have the capacity to set the stage, harness 
resources, and create a common agenda. Commitment to high 
performance requires the contributions of many players. ❖ 



The Mission of NCSALL 

The National Center for the Study of Adult Learning and Literacy (NCSALL) pursues basic and applied research in the field of adult 
basic education; builds partnerships between researchers and practitioners; disseminates research and best practices to practitioners, 
scholars, and policymakers; and works with the field to develop a comprehensive research agenda. 

NCSALL is a collaborative effort between the Harvard Graduate School of Education and World Education. The University of 
Tennessee, Portland State University and Rutgers University are NCSALL’s partners. One more partner in the Mid-west will be added 
in the future. NCSALL is funded by the Educational Research and Development Centers Program, Award Number R309B60(X)2, as 
administered by the Office of Educational Research and Improvement/National Institute of Postsecondary Education, Libraries and 
Lifelong Learning, U.S. Department of Education. 

Full Report Available 

To receive a copy of the full report, write to Kimberly French at NCSALL Reports, World Education, 44 Farnsworth Street, Boston, 
MA 02210-121 1. Please be sure to include a check in the amount of $10.00 made out to World Education. For more information about 
other NCSALL publications, call (617) 482-9485 or e-mail us at NCSALL@WorldEd.org. Visit our web site at 
http://hugsel.harvard.edu/-ncsall. 
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